RuMedBench: A Russian Medical Language Understanding Benchmark

The paper describes the open Russian medical language understanding benchmark covering several task types (classification, question answering, natural inference, named entity recognition) on a number of novel text sets. Given sensitive nature data in healthcare, such partially closes problem dataset absence. We prepare unified format labeling, split, and evaluation metrics for new tasks. remaining tasks are from existing datasets with few modifications. A single-number metric expresses model's ability to cope benchmark. Moreover, we implement baseline models, simple ones neural networks transformer architecture, release code. Expectedly, more advanced models yield better performance, but even model is enough decent result some Furthermore, all tasks, provide human evaluation. Interestingly outperform humans large-scale classification However, advantage intelligence remains requiring knowledge reasoning.